Skip to content

fix: post setup awareness once on PR open#76

Merged
dkargatzis merged 2 commits into
mainfrom
fix/one-time-pr-setup-awareness
Jul 13, 2026
Merged

fix: post setup awareness once on PR open#76
dkargatzis merged 2 commits into
mainfrom
fix/one-time-pr-setup-awareness

Conversation

@dkargatzis

@dkargatzis dkargatzis commented Jul 13, 2026

Copy link
Copy Markdown
Member

Posts the missing-rules setup awareness comment only once, when a PR is initially opened. Prevents duplicate comments from later pushes, reviews, and thread events while preserving neutral check-run guidance.

Summary by CodeRabbit

  • New Features

    • Setup-awareness comments are now posted only when a newly opened pull request lacks rules.
    • Duplicate setup-awareness comments are prevented, including during concurrent events.
    • Comment history is fully reviewed across paginated results for improved acknowledgment and notification handling.
  • Bug Fixes

    • Later commits, reviews, and re-runs refresh checks without repeating setup comments.
    • Improved detection of existing Watchflow comments and comment-related failures.
  • Documentation

    • Updated contributor, onboarding, feature, and overview documentation to reflect the revised behavior.

Signed-off-by: Dimitris Kargatzis <dkargatzis@gmail.com>
@dkargatzis dkargatzis self-assigned this Jul 13, 2026
@coderabbitai

coderabbitai Bot commented Jul 13, 2026

Copy link
Copy Markdown

Review Change Stack

Warning

Review limit reached

@dkargatzis, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 2 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: e48dcbeb-980f-4a3a-b343-ace15ff5c4f1

📥 Commits

Reviewing files that changed from the base of the PR and between e642783 and 43959a4.

📒 Files selected for processing (8)
  • docs/concepts/overview.md
  • docs/features.md
  • docs/getting-started/quick-start.md
  • src/event_processors/pull_request/processor.py
  • src/integrations/github/check_runs.py
  • src/presentation/github_formatter.py
  • tests/unit/event_processors/test_pull_request_processor.py
  • tests/unit/presentation/test_github_formatter.py
📝 Walkthrough

Walkthrough

Watchflow now posts a marked setup-awareness comment only when an initially opened pull request lacks rules, prevents duplicate concurrent posts, paginates issue comments, reuses comment snapshots, and updates documentation and tests for the revised behavior.

Changes

Setup-awareness comment flow

Layer / File(s) Summary
Setup-awareness comment marker
src/presentation/github_formatter.py, tests/unit/presentation/test_github_formatter.py
Rules-not-configured comments now begin with a hidden deduplication marker, with unit coverage for the generated prefix.
Comment retrieval and acknowledgment parsing
src/integrations/github/api.py, src/event_processors/pull_request/enricher.py, tests/unit/integrations/github/test_api.py
Issue comments are paginated, failures return None, and acknowledgment parsing is extracted into a dedicated helper.
Onboarding and deduplication flow
src/event_processors/pull_request/processor.py, tests/unit/event_processors/test_pull_request_processor.py
The processor reuses one comment snapshot, recognizes Watchflow-authored markers, limits setup comments to opened PRs, and serializes concurrent posting.
Documented onboarding behavior
CONTRIBUTING.md, README.md, docs/concepts/overview.md, docs/features.md, docs/getting-started/quick-start.md, docs/index.md
Documentation now describes the one-time setup-awareness comment and subsequent neutral check refreshes.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant PullRequestProcessor
  participant GitHubClient
  participant PullRequestEnricher
  PullRequestProcessor->>GitHubClient: Fetch paginated issue comments
  GitHubClient-->>PullRequestProcessor: Return comment snapshot
  PullRequestProcessor->>PullRequestEnricher: Parse acknowledgments
  PullRequestEnricher-->>PullRequestProcessor: Return acknowledgment map
  PullRequestProcessor->>PullRequestProcessor: Check action and managed markers
  PullRequestProcessor->>GitHubClient: Create setup-awareness comment when needed
Loading

Possibly related PRs

Poem

I’m a rabbit hopping through the check,
Marking comments so duplicates stay in check.
One setup note when new PRs appear,
Then neutral updates hop along clear.
Paginated comments, neatly read—
“Binky!” says the bot, “the flow is fed.”

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 43.18% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title matches the main change: posting a setup-awareness comment once, only when a PR is opened.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/one-time-pr-setup-awareness

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@watchflow

watchflow Bot commented Jul 13, 2026

Copy link
Copy Markdown

🛡️ Watchflow Governance Checks

Status: ❌ 3 Violations Found

🟡 Medium Severity (3)

Checks PR description (body) and title for a linked issue reference (e.g. #123, Fixes #123, Closes #456). Use when the rule requires issue refs in either field.

PR does not reference a linked issue (e.g. #123 or closes #123 in body/title)
How to fix: Add an issue reference in the PR title or description (e.g. Fixes #123).

Validates if the PR description meets minimum length requirements

PR description is empty
How to fix: Add a description with at least 50 characters.

Ensures PRs that modify source code also include a CHANGELOG or .changeset addition.

Source code was modified without a corresponding CHANGELOG update.
How to fix: Add an entry to CHANGELOG.md or generate a new .changeset file describing your changes.


💡 Reply with @watchflow ack [reason] to override these rules, or @watchflow help for commands.

Thanks for using Watchflow! It's completely free for OSS and private repositories. You can also self-host it easily.

@codecov-commenter

codecov-commenter commented Jul 13, 2026

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 93.85797% with 32 lines in your changes missing coverage. Please review.

❌ Your project status has failed because the head coverage (73.9%) is below the target coverage (80.0%). You can increase the head coverage or adjust the target coverage.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

@@           Coverage Diff           @@
##            main     #76     +/-   ##
=======================================
+ Coverage   73.0%   73.9%   +0.8%     
=======================================
  Files        181     181             
  Lines      13481   13903    +422     
=======================================
+ Hits        9851   10276    +425     
+ Misses      3630    3627      -3     

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 60ae336...43959a4. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
docs/features.md (1)

80-90: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Use the new setup-awareness terminology consistently.

This section still calls the posted comment a “welcome comment” while the updated behavior is described as a “setup-awareness comment.” Rename the bullet to avoid implying two different comment types.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/features.md` around lines 80 - 90, In the missing-rules behavior
description, update the bullet label “welcome comment” to “setup-awareness
comment” so the terminology matches the later reference and describes a single
comment type.
🧹 Nitpick comments (2)
src/integrations/github/api.py (1)

981-1004: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick win

Add a pagination cap to get_issue_comments.

The while True loop has no upper bound on page/total comments fetched. A PR with an unusually large comment thread (e.g. spam or a very active discussion) will trigger many sequential GitHub API round-trips inside a single webhook-processing task, delaying check-run/comment posting and burning API rate limit. Consider capping total pages/comments and logging when the cap is hit.

♻️ Proposed fix
             session = await self._get_session()
             comments: list[dict[str, Any]] = []
             page = 1
+            max_pages = 50  # ~5000 comments; guards against pathological threads
             while True:
                 async with session.get(url, headers=headers, params={"per_page": 100, "page": page}) as response:
                     if response.status != 200:
                         ...
                     result = await response.json()
                     ...
                     page_comments = cast("list[dict[str, Any]]", result)
                     comments.extend(page_comments)
                     if len(page_comments) < 100:
                         logger.info(f"Retrieved {len(comments)} comments for issue #{issue_number} in {repo}")
                         return comments
                     page += 1
+                    if page > max_pages:
+                        logger.warning(
+                            f"Reached max_pages={max_pages} fetching comments for issue #{issue_number} in {repo}; "
+                            "results may be truncated"
+                        )
+                        return comments
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/integrations/github/api.py` around lines 981 - 1004, Update
get_issue_comments to enforce an upper bound on pagination, using a clear
maximum page or total-comment cap before continuing requests. When the cap is
reached, log that pagination was truncated and return the comments collected so
far, while preserving the existing response validation and normal completion
behavior.
src/event_processors/pull_request/processor.py (1)

355-420: 🩺 Stability & Availability | 🔵 Trivial

Local lock only serializes within one process.

The reference-counted asyncio.Lock correctly serializes concurrent async tasks within a single worker instance (verified against the concurrency test), and the GitHub comment-list check (_has_setup_awareness_comment) provides a secondary cross-process safety net. However, if this processor runs across multiple worker instances/pods, there is still a TOCTOU window between the get_issue_comments check and create_pull_request_comment across processes — two different workers could both pass the check before either posts. Worth confirming whether delivery-ID dedup upstream (per CONTRIBUTING.md) also covers "different but overlapping" events on the same PR across workers, or only identical redeliveries.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/event_processors/pull_request/processor.py` around lines 355 - 420,
Confirm whether upstream delivery-ID deduplication prevents overlapping,
distinct events for the same PR across worker instances, not only identical
redeliveries. If it does not, replace or augment the local
_setup_awareness_locks coordination in _post_rules_not_configured_comment with a
shared cross-process mechanism that atomically guards the comment existence
check and creation, while preserving the existing opened-event and
single-comment behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@docs/features.md`:
- Around line 80-90: In the missing-rules behavior description, update the
bullet label “welcome comment” to “setup-awareness comment” so the terminology
matches the later reference and describes a single comment type.

---

Nitpick comments:
In `@src/event_processors/pull_request/processor.py`:
- Around line 355-420: Confirm whether upstream delivery-ID deduplication
prevents overlapping, distinct events for the same PR across worker instances,
not only identical redeliveries. If it does not, replace or augment the local
_setup_awareness_locks coordination in _post_rules_not_configured_comment with a
shared cross-process mechanism that atomically guards the comment existence
check and creation, while preserving the existing opened-event and
single-comment behavior.

In `@src/integrations/github/api.py`:
- Around line 981-1004: Update get_issue_comments to enforce an upper bound on
pagination, using a clear maximum page or total-comment cap before continuing
requests. When the cap is reached, log that pagination was truncated and return
the comments collected so far, while preserving the existing response validation
and normal completion behavior.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 849fe2e7-d586-4660-bb61-c79b8eb10476

📥 Commits

Reviewing files that changed from the base of the PR and between 60ae336 and e642783.

📒 Files selected for processing (13)
  • CONTRIBUTING.md
  • README.md
  • docs/concepts/overview.md
  • docs/features.md
  • docs/getting-started/quick-start.md
  • docs/index.md
  • src/event_processors/pull_request/enricher.py
  • src/event_processors/pull_request/processor.py
  • src/integrations/github/api.py
  • src/presentation/github_formatter.py
  • tests/unit/event_processors/test_pull_request_processor.py
  • tests/unit/integrations/github/test_api.py
  • tests/unit/presentation/test_github_formatter.py
📜 Review details
⚠️ CI failures not shown inline (1)

GitHub Check: Watchflow Rules: 3 rule violations found

Conclusion: failure

View job details

🚨 3 violations found: 3 medium
# Watchflow Rule Violations
<details>
<summary><b>🟡 Medium Severity (3)</b></summary>
### Checks PR description (body) and title for a linked issue reference (e.g. `#123`, Fixes `#123`, Closes `#456`). Use when the rule requires issue refs in either field.
PR does not reference a linked issue (e.g. `#123` or closes `#123` in body/title)
**How to fix:** Add an issue reference in the PR title or description (e.g. Fixes `#123`).
### Validates if the PR description meets minimum length requirements
PR description is empty
**How to fix:** Add a description with at least 50 characters.
### Ensures PRs that modify source code also include a CHANGELOG or .changeset addition.
Source code was modified without a corresponding CHANGELOG update.
**How to fix:** Add an entry to CHANGELOG.md or generate a new .changeset file describing your changes.
</details>
---
💡 *To configure rules, edit the `.watchflow/rules.yaml` file in this repository.*
🧰 Additional context used
📓 Path-based instructions (3)
docs/**

📄 CodeRabbit inference engine (.cursor/rules/guidelines.mdc)

docs/**: Update README and docs for user-visible changes and migrations
Provide runnable examples for new/changed rules; keep cross-links current

Files:

  • docs/index.md
  • docs/features.md
  • docs/getting-started/quick-start.md
  • docs/concepts/overview.md
**/*.py

📄 CodeRabbit inference engine (.cursor/rules/guidelines.mdc)

**/*.py: Use modern typing only: dict[str, Any], list[str], str | None (no Dict, List, Optional)
GitHub/HTTP/DB calls must be async def; avoid blocking calls (time.sleep, sync HTTP) in async paths
All agent outputs and external payloads must use validated BaseModel from Pydantic
Use dataclasses for internal immutable state where appropriate
Use structured logging at boundaries with fields: operation, subject_ids, decision, latency_ms
Implement Agent pattern: single-responsibility agents with typed inputs/outputs
Use Decorator pattern for retries, metrics, caching as cross-cutting concerns
Agent outputs must include: decision, confidence (0..1), short reasoning, recommendations, strategy_used
Implement confidence policy: reject or route to human-in-the-loop when confidence < 0.5
Use minimal, step-driven prompts; provide Chain-of-Thought only for complexity > 0.7 or ambiguity > 0.6
Strip secrets/PII from agent prompts; scope tools; keep raw reasoning out of logs (store summaries only)
Cache idempotent lookups; lazy-import heavy dependencies; bound fan-out with asyncio.Semaphore
Avoid redundant LLM calls; memoize per event when safe
Use domain errors (e.g., AgentError) with error_type, message, context, timestamp, retry_count
Use exponential backoff for transient failures; circuit-break noisy integrations when needed
Fail closed for risky decisions; provide actionable remediation in error paths
Validate all external inputs; verify webhook signatures
Implement prompt-injection hardening; sanitize repository content passed to LLMs
Performance targets: Static validation ~<100ms typical, hybrid decisions sub-second when cache warm, budget LLM paths thoughtfully
Reject old typing syntax (Dict, List, Optional) in code review
Reject blocking calls in async code; reject bare except: clauses; reject swallowed errors
Reject LLM calls for trivial/deterministic checks
Reject unvalidated agent outputs and missing confidenc...

Files:

  • tests/unit/integrations/github/test_api.py
  • tests/unit/presentation/test_github_formatter.py
  • src/presentation/github_formatter.py
  • src/integrations/github/api.py
  • src/event_processors/pull_request/enricher.py
  • src/event_processors/pull_request/processor.py
  • tests/unit/event_processors/test_pull_request_processor.py
tests/**/*.py

📄 CodeRabbit inference engine (.cursor/rules/guidelines.mdc)

tests/**/*.py: Write unit tests for deterministic rule evaluation (pass/warn/block), model validation, and error paths
Write integration tests for webhook parsing, idempotency, multi-agent coordination, and state persistence
Use pytest.mark.asyncio for async tests; avoid live network calls; freeze time and seed randomness
Write regression tests for every bug fix; keep CI coverage thresholds green

Files:

  • tests/unit/integrations/github/test_api.py
  • tests/unit/presentation/test_github_formatter.py
  • tests/unit/event_processors/test_pull_request_processor.py
🧠 Learnings (1)
📚 Learning: 2026-03-27T12:52:44.067Z
Learnt from: oleksii-quinta
Repo: warestack/watchflow PR: 67
File: src/webhooks/handlers/issue_comment.py:153-159
Timestamp: 2026-03-27T12:52:44.067Z
Learning: When enqueuing processor tasks using `task_queue`, follow the documented pre-built-task pattern: (1) build the task with `pre_built_task = task_queue.build_task(event_type, payload, processor.process, delivery_id=...)`; (2) call `task_queue.enqueue(processor.process, event_type, payload, pre_built_task, delivery_id=...)` by passing the pre-built task as a single positional `*args` element; (3) ensure the worker ultimately calls `await processor.process(pre_built_task)` (i.e., the processor `process(self, task: Task)` receives the `Task` instance). This matches the expectation that `enqueue` stores the pre-built task in the wrapper Task’s `args` as described by `build_task`’s docstring (“pass as single arg to enqueue”).

Applied to files:

  • src/presentation/github_formatter.py
  • src/integrations/github/api.py
  • src/event_processors/pull_request/enricher.py
  • src/event_processors/pull_request/processor.py
🔇 Additional comments (12)
src/presentation/github_formatter.py (1)

7-8: LGTM!

Also applies to: 177-177

tests/unit/presentation/test_github_formatter.py (1)

3-6: LGTM!

Also applies to: 91-94

README.md (1)

137-137: LGTM!

docs/concepts/overview.md (1)

28-28: LGTM!

docs/getting-started/quick-start.md (1)

10-10: LGTM!

Also applies to: 29-37

docs/index.md (1)

47-47: LGTM!

src/event_processors/pull_request/enricher.py (1)

112-135: LGTM!

tests/unit/integrations/github/test_api.py (1)

116-140: LGTM!

src/event_processors/pull_request/processor.py (1)

1-11: LGTM!

Also applies to: 21-21, 32-32, 161-161, 210-216, 270-270, 304-306, 322-326, 442-502

tests/unit/event_processors/test_pull_request_processor.py (2)

25-25: LGTM!

Also applies to: 37-59, 68-68, 128-151, 188-191, 250-252, 286-321, 323-385, 386-407


1-12: 🎯 Functional Correctness

Task import is present tests/unit/event_processors/test_pull_request_processor.py:13 already imports Task, so the MagicMock(spec=Task) usages are valid.

			> Likely an incorrect or invalid review comment.
CONTRIBUTING.md (1)

10-10: LGTM!

@watchflow

watchflow Bot commented Jul 13, 2026

Copy link
Copy Markdown

🛡️ Watchflow Governance Checks

Status: ❌ 2 Violations Found

🟡 Medium Severity (2)

Checks PR description (body) and title for a linked issue reference (e.g. #123, Fixes #123, Closes #456). Use when the rule requires issue refs in either field.

PR does not reference a linked issue (e.g. #123 or closes #123 in body/title)
How to fix: Add an issue reference in the PR title or description (e.g. Fixes #123).

Ensures PRs that modify source code also include a CHANGELOG or .changeset addition.

Source code was modified without a corresponding CHANGELOG update.
How to fix: Add an entry to CHANGELOG.md or generate a new .changeset file describing your changes.


💡 Reply with @watchflow ack [reason] to override these rules, or @watchflow help for commands.

Thanks for using Watchflow! It's completely free for OSS and private repositories. You can also self-host it easily.

Signed-off-by: Dimitris Kargatzis <dkargatzis@gmail.com>
@watchflow

watchflow Bot commented Jul 13, 2026

Copy link
Copy Markdown

🛡️ Watchflow Governance Checks

Status: ❌ 3 Violations Found

🟡 Medium Severity (3)

Checks PR description (body) and title for a linked issue reference (e.g. #123, Fixes #123, Closes #456). Use when the rule requires issue refs in either field.

PR does not reference a linked issue (e.g. #123 or closes #123 in body/title)
How to fix: Add an issue reference in the PR title or description (e.g. Fixes #123).

Validates that total lines changed (additions + deletions) in a PR do not exceed a maximum; enforces a maximum LOC per pull request.

Pull request exceeds maximum lines changed (1090 > 500)
How to fix: Reduce the size of this PR to at most 500 lines changed (additions + deletions).

Ensures PRs that modify source code also include a CHANGELOG or .changeset addition.

Source code was modified without a corresponding CHANGELOG update.
How to fix: Add an entry to CHANGELOG.md or generate a new .changeset file describing your changes.


💡 Reply with @watchflow ack [reason] to override these rules, or @watchflow help for commands.

Thanks for using Watchflow! It's completely free for OSS and private repositories. You can also self-host it easily.

@dkargatzis dkargatzis merged commit 302a326 into main Jul 13, 2026
6 of 7 checks passed
@dkargatzis dkargatzis deleted the fix/one-time-pr-setup-awareness branch July 13, 2026 16:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants